Reinforcement Learning Dynamics in Social Dilemmas

In this paper we replicate and advance Macy and Flache\'s (2002; Proc. Natl. Acad. Sci. USA, 99, 7229–7236) work on the dynamics of reinforcement learning in 2�2 (2-player 2-strategy) social dilemmas. In particular, we provide further insight into the solution concepts that they describe, illustrate some recent analytical results on the dynamics of their model, and discuss the robustness of such results to occasional mistakes made by players in choosing their actions (i.e. trembling hands). It is shown here that the dynamics of their model are strongly dependent on the speed at which players learn. With high learning rates the system quickly reaches its asymptotic behaviour; on the other hand, when learning rates are low, two distinctively different transient regimes can be clearly observed. It is shown that the inclusion of small quantities of randomness in players\' decisions can change the dynamics of the model dramatically.Reinforcement Learning; Replication; Game Theory; Social Dilemmas; Agent-Based; Slow Learning

"Test two, choose the better" leads to high cooperation in the Centipede game

Explaining cooperative experimental evidence in the Centipede game constitutes a challenge for rational game theory. Traditional analyses of Centipede based on backward induction predict uncooperative behavior. Furthermore, analyses based on learning or adaptation under the assumption that those strategies that are more successful in a population tend to spread at a higher rate usually make the same prediction. In this paper we consider an adaptation model in which agents in a finite population do adopt those strategies that turn out to be most successful, according to their own experience. However, this behavior leads to an equilibrium with high levels of cooperation and whose qualitative features are consistent with experimental evidence.Financial support from the Spanish State Research Agency (PID2020-118906GB-I00 / AEI / 10.13039/501100011033), from “Junta de Castilla y León - Consejería de Educación” through BDNS 425389, from the Spanish Ministry of Science, Innovation and Universities (PRX18-00182, PRX19/00113), and from the Fulbright Program (PRX19/00113), is gratefully acknowledged

Forecasting VARMA processes using VAR models and subspace-based state space models

VAR modelling is a frequent technique in econometrics for linear processes. VAR modelling offers some desirable features such as relatively simple procedures for model specification (order selection) and the possibility of obtaining quick non-iterative maximum likelihood estimates of the system parameters. However, if the process under study follows a finite-order VARMA structure, it cannot be equivalently represented by any finite-order VAR model. On the other hand, a finite-order state space model can represent a finite-order VARMA process exactly, and, for state-space modelling, subspace algorithms allow for quick and non-iterative estimates of the system parameters, as well as for simple specification procedures. Given the previous facts, we check in this paper whether subspace-based state space models provide better forecasts than VAR models when working with VARMA data generating processes. In a simulation study we generate samples from different VARMA data generating processes, obtain VAR-based and state-space-based models for each generating process and compare the predictive power of the obtained models. Different specification and estimation algorithms are considered; in particular, within the subspace family, the CCA (Canonical Correlation Analysis) algorithm is the selected option to obtain state-space models. Our results indicate that when the MA parameter of an ARMA process is close to 1, the CCA state space models are likely to provide better forecasts than the AR models. We also conduct a practical comparison (for two cointegrated economic time series) of the predictive power of Johansen restricted-VAR (VEC) models with the predictive power of state space models obtained by the CCA subspace algorithm, including a density forecasting analysis.subspace algorithms; VAR; forecasting; cointegration; Johansen; CCA

Techniques to Understand Computer Simulations: Markov Chain Analysis

The aim of this paper is to assist researchers in understanding the dynamics of simulation models that have been implemented and can be run in a computer, i.e. computer models. To do that, we start by explaining (a) that computer models are just input-output functions, (b) that every computer model can be re-implemented in many different formalisms (in particular in most programming languages), leading to alternative representations of the same input-output relation, and (c) that many computer models in the social simulation literature can be usefully represented as time-homogeneous Markov chains. Then we argue that analysing a computer model as a Markov chain can make apparent many features of the model that were not so evident before conducting such analysis. To prove this point, we present the main concepts needed to conduct a formal analysis of any time-homogeneous Markov chain, and we illustrate the usefulness of these concepts by analysing 10 well-known models in the social simulation literature as Markov chains. These models are: • Schelling\'s (1971) model of spatial segregation • Epstein and Axtell\'s (1996) Sugarscape • Miller and Page\'s (2004) standing ovation model • Arthur\'s (1989) model of competing technologies • Axelrod\'s (1986) metanorms models • Takahashi\'s (2000) model of generalized exchange • Axelrod\'s (1997) model of dissemination of culture • Kinnaird\'s (1946) truels • Axelrod and Bennett\'s (1993) model of competing bimodal coalitions • Joyce et al.\'s (2006) model of conditional association In particular, we explain how to characterise the transient and the asymptotic dynamics of these computer models and, where appropriate, how to assess the stochastic stability of their absorbing states. In all cases, the analysis conducted using the theory of Markov chains has yielded useful insights about the dynamics of the computer model under study.Computer Modelling, Simulation, Markov, Stochastic Processes, Analysis, Re-Implementation

Best experienced payoff dynamics and cooperation in the centipede game

We study population game dynamics under which each revising agent tests each of his strategies a fixed number of times, with each play of each strategy being against a newly drawn opponent, and chooses the strategy whose total payoff was highest. In the centipede game, these best experienced payoff dynamics lead to cooperative play. When strategies are tested once, play at the almost globally stable state is concentrated on the last few nodes of the game, with the proportions of agents playing each strategy being largely independent of the length of the game. Testing strategies many times leads to cyclical play.U.S. National Science Foundation (Grants SES-1458992 and SES- 1728853), the U.S. Army Research Office (Grants W911NF-17-1-0134 MSN201957), Project ECO2017-83147- C2-2-P (MINECO/AEI/FEDER, UE), and the Spanish Ministerio de Educación, Cultura, y Deporte (Grants PRX15/00362 and PRX16/00048

EvoDyn-3s: A Mathematica computable document to analyze evolutionary dynamics in 3-strategy games

EvoDyn-3s generates phase portraits of evolutionary dynamics, as well as data for the analysis of their equilibria. The considered evolutionary dynamics are ordinary differential equations based on adaptive processes taking place in a population of players who are randomly and repeatedly matched in couples to play a 2-player symmetric normal-form game with three strategies. EvoDyn-3s calculates the rest points of the dynamics using exact arithmetic, and represents them. It also provides the eigenvalues of the Jacobian of the dynamics at the isolated rest points, which are useful to evaluate their local stability. The user only needs to specify the 3 × 3 payoff matrix of the game and choose the dynamics.Spanish Ministry of Science and Innovation ’s project ECO2017-83147-C2-2-P (MINECO/AEI/FEDER, UE

Teaching the mean-field approximation

[ENG] Many University subjects that we teach in engineering degrees (e.g. Control Theory, Operations Research, Game Theory, Physics, Fluid Dynamics, Modelling and Control of Queuing and Production Systems, Stochastic Processes, and Network Theory) make extensive use of the so-called mean-field analysis. In these subjects, the mean-field analysis often appears as an approximation of a discrete time stochastic process where a) the inherent stochasticity of the original process is replaced with determinism, and b) the time discreteness of the original process is replaced with time continuity. Thus, the mean-field approximation is presented as a continuous time differential equation that can approximate the dynamics of the discrete time stochastic process under investigation. In this paper we present a teaching methodology that we have found useful for introducing students to the mean-field analysis, and we provide some accompanying teaching material –in the form of computer models– that other academics may want to use in their own lecturesThe authors gratefully acknowledge financial support from the Spanish JCyL (VA006B09, GR251/2009), MICINN (SICOSSYS: TIN2008-06464-C03-02; CONSOLIDER-INGENIO 2010: CSD2010-00034; DPI2010-16920) and L.R.I. from the Spanish Ministry of Education (JC2009-00263)

Errors and Artefacts in Agent-Based Modelling

The objectives of this paper are to define and classify different types of errors and artefacts that can appear in the process of developing an agent-based model, and to propose activities aimed at avoiding them during the model construction and testing phases. To do this in a structured way, we review the main concepts of the process of developing such a model – establishing a general framework that summarises the process of designing, implementing, and using agent-based models. Within this framework we identify the various stages where different types of errors and artefacts may appear. Finally we propose activities that could be used to detect (and hence eliminate) each type of error or artefact.Verification, Replication, Artefact, Error, Agent-Based Modelling, Modelling Roles

Mixing and diffusion in a two-type population

The outbreak of epidemics, the rise of religious radicalization or the motivational influence of fellow students in classrooms are some of the issues that can be described as diffusion processes in heterogeneous groups. Understanding the role that interaction patterns between groups (e.g. homophily or segregation) play in the diffusion of certain traits or behaviours is a major challenge for contemporary societies. Here, we study the impact on diffusion processes of mixing (or, alternatively, segregating) two groups that present different sensitivities or propensities to contagion. We find non-monotonic effects of mixing and inefficient segregation levels, i.e. situations where a change in the mixing level can benefit both groups, e.g. where an increase in the mixing level can reduce the expected contagion levels in both groups. These findings can have fundamental consequences for the design of inclusion policies.D.L.-P. from the Spanish Ministry of Science and Innovation (ECO2011-22919) and from project ECO2017-83147-C2-1-P (MINECO/AEI/FEDER, UE). L.R.I. and S.S.I. from project ECO2017-83147-C2-2-P (MINECO/AEI/FEDER, UE